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Summary:  New  information  technology  promises 
more  information  and  advanced  automated  functions  in 
future  cockpits  of  military  aircraft.  However  the 
cognitive  human  capabilities  stay  the  same.  This  may 
result  in  an  overload  of  the  human  pilot.  Cognitive 
assistant  systems  are  being  developed  to  compensate 
for  this  mismatch.  This  paper  introduces  principals  of 
cognitive  systems  which  exhibit  human-like 
capabilities  as  interpretation  and  diagnosis  of  the 
situation,  planning  and  decision  making.  Furthermore, 
CAMA  (Crew  Assistant  Military  Aircraft),  a prototype 
of  a cognitive  assistant  system,  will  be  introduced. 
CAMA’s  functionality  will  be  shown  and  some  results 
from  flight  simulator  test  runs  will  be  presented. 

Motivation:  Environment  and  scenarios  of  military 
transport  missions  have  changed  over  the  last  few 
years  and  will  definitely  undergo  even  more  changes  in 
the  next  decade.  New  information  technology, 
including  telecommunication  as  well  as  hardware 
which  is  continuously  growing  more  powerful  will  find 
its  way  into  the  future  military  aircraft.  Online  data  of 
upcoming  threats,  detailed  weather  information,  terrain 
data  and  knowledge  about  weapon  systems  will  be 
available.  Combined  air  operations  with  participation 
of  AWACS,  fighters,  bombers  and  transport  aircraft  are 
likely  with  the  need  for  more  communication. 

There  is  a rising  amount  of  mission-relevant 
information,  that  has  to  be  processed  by  the  human 
operator  who  is  also  in  charge  of  flying  the  aircraft. 
Considering  the  complexity  and  manifold  of 
automation  in  current  cockpits  and  even  more  in  those 
of  the  next  generation  it  can  clearly  be  seen,  that  it  will 
become  more  and  more  difficult  for  the  human  to  keep 
situation  awareness  and  perform  all  the  tasks  in  an 
efficient  way  without  errors. 

This  leads  to  the  central  question: 

How  can  we  make  the  best  use  of  the  potential  given  by 
the  new  technologies  without  overloading  the  cognitive 
capabilities  of  the  human  operator  ? 


There  is  an  approach  where  automation  in  the 
conventional  way  is  being  added  to  the  cockpit,  hoping 
for  increased  productivity  and  effectiveness.  As  we 
know  from  experience,  however  the  conventional 
automation  can  increase  safety  only  up  to  a certain 
level.  Further  increase  of  complexity  in  the 
conventional  way  can  lead  to  a safety  decline  as  shown 
in  figure  1 m its  principal  relationships. 


Productivity  Safety 

Cognitive  Automation 


Complexity  of  Automation  Complexity  of  Automation 

Figure  1 : The  Effect  of  Conventional  and 
Cognitive  Automation  on  Productivity 
and  Safety 

Recent  accidents  of  commercial  aircraft  with  state-of- 
the-art  “conventional  cockpit  automation”  provided 
sufficient  evidence  for  this  particular  consequence.  [1] 
identifies  besides  complexity  as  such  also  other  design 
elements  more  or  less  as  part  of  complexity  like 
coupling  of  automated  features, 
autonomy  with  unexpected  self  initiated  machine 
behaviour  and 
inadequate  feedback 

which  are  typical  causes  for  respective  mishaps.  In 
military  aviation  the  situation  can  be  expected  to  be 
critical  due  to  permanently  increasing  requirements  for 
information  processing. 

Cognitive  automation:  How  can  this  situation  be  dealt 
with?  The  critical  point  is,  how  automation  can  be  done 
in  an  effective  manner.  Automation  should  not  be  a 
replacement  for  the  pilot,  but  instead  should  work  in  a 
cooperative  way  with  the  pilot.  In  the  ideal  case  it 
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should  work  like  a kind  of  “electronic  crewmember”, 
with  the  cognitive  capabilities  like  those  of  the  human, 
but  without  all  its  possible  deficiencies. 

[2]  postulates  basic  design  requirements  founded  on 
these  cognitive  capabilities: 

Requirement  (1)  is  to  avoid  failings  in  situation 
awareness  and  reads: 

It  must  be  ensured  along  with  the  representation  of  the 
full  picture  of  the  flight  situation  that  the  attention  of 
the  cockpit  crew  is  guided  towards  the  objectively  most 
urgent  task  or  sub-task  as  demanded  in  that  situation. 

Requirement  (2)  is  to  avoid  overcharge  in  decision 
making/planning/plan  execution  and  reads: 

Situation  awareness  might  ha\>e  been  achieved  and  still 
a situation  with  overcharge  of  the  cockpit  crew  might 
come  up.  In  this  case  the  situation  has  to  be 
automatically  transferred  into  a situation  which  can  be 
handled  by  the  crew  in  a normal  manner. 

“Cognitive  automation”  is  the  only  way  to  ensure 
increase  of  productivity  through  automation  without 
loss  of  safety  (see  fig.  1). 

The  difference  between  cognitive  and  conventional 
automation  can  also  be  illustrated  by  Rasmussen’s 
scheme  of  human  cognitive  behaviour 

[3] . 


Figure  2:  Conventional  vs.  cognitive  Automation  [4] 


As  figure  2 shows,  conventional  automation  covers 
nearly  the  whole  of  skill  based  human  behaviour.  The 
rule  based  behaviour  can  only  partly  be  covered,  and 
on  the  knowledge  based  level  only  planning 
calculations  can  be  provided  by  conventional 
automation. 

Cognitive  automation  comprises  the  entire  rule  and 
knowledge  based  level,  as  well  as  the  skill  based  level, 
thereby  giving  the  system  human  like  capabilities  to: 
independently  assess  the  current  goals  of  the  crew', 
as  well  as  information  about  the  aircraft,  the 
environment  including  the  tactical  situation,  the 
weapon  systems  and  the  aircrew  activities 
- understand  the  flight  situation  by  independently 
interpreting  the  situation  subject  to  the  goals 


- detect  the  pilot's  intents  and  possible  errors 

- detect  possible  conflicts  of  current  plans  but  also 
the  opportunities  arising  from  the  changing 
environment 

- know  which  information  the  crew  needs 

- support  necessary  re-planning  and  decision  making 

- initiate  a natural,  human-like  communication  to 
match  the  system’s  internal  pictures  of  the  situation 
with  those  of  the  pilot. 

The  symbiosis  of  cognitive  automation  combined  with 
the  strength  of  the  human  will  lead  to  a more  efficient 
and  safer  mission  execution 


The  Cognitive  Process:  To  realise  the  cognitive 
approach  as  a technical  process  human  cognition 
provides  a good  guideline.  The  following  core 
elements  can  be  identified: 

Situation  monitoring 
(perception  and  interpretation) 

Diagnosis  of  the  situation 

Decision  making  and/or  planning 
Execution/acti  vation 


They  are  forming  the  cognitive  loop  as  shown  in  figure 
3.  The  environment  of  the  cognitive  assistant,  which  is 
named  the  real  world,  presents  stimuli,  which  can  be 
detected  by  different  kinds  of  sensory  systems.  Both 
the  environmental  stimuli  outside  and  inside  the 
cockpit  are  taken  into  account.  This  represents  the 
situation  monitoring  element,  which  comprises  the 
process  of  perception  of  all  relevant  situation  features. 
This  is  closely  interrelated  with  the  process  of  situation 
analysis  in  order  to  achieve  a certain  level  of 
abstraction,  thereby  establishing  situation-relevant 
“objects”  which  help  to  understand  what  is  different 


Figure  3:  The  Cognitive  Loop 


between  the  expected  and  actual  situation.  These 
differences  are  dealt  with  in  a higher  level  of 
abstraction  by  the  so-called  situation  diagnosis  process. 
The  differences  are  evaluated  against  given  objectives, 
the  relevant  goals , which  are  known  to  be  pursued 
during  the  mission,  and  which  are  the  same  the  aircrew 
has  in  mind.  Only  the  knowledge  about  these  goals 
makes  situation  awareness  possible  in  the  technical 
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cognitive  loop.  Thereby  conflicts  and/or  opportunities 
may  be  detected  which  may  call  for  immediate  actions 
or  some  flight  plan  changes. 

In  the  latter  case  a planning  process  is  activated  to 
generate  alternatives  for  interim-goals,  plans,  and 
actions.  In  compliance  with  the  given  overall 
objectives,  the  most  appropriate  ones  are  chosen  for 
proposals. 

Concerning  the  assistant  system,  the  execution  element 
of  the  cognitive  process  plays  a very  central  and 
important  role,  as  it  includes  the  communication  with 
the  crew.  It  is  carried  it  out  on  the  basis  of  profound 
internal  knowledge  about  what  information  the  aircrew 
is  looking  for,  why  and  when.  On  the  other  hand,  the 
crew  should  at  any  time  be  able  to  ask  for  certain 
information  within  the  system.  A sophisticated  MMI  is 
required  to  accomplish  this  task. 

It  is  also  taken  into  consideration  that  the  crew  may 
react  different  compared  to  the  systems  proposals, 
because  certain  factors  were  not  taken  into 
consideration  at  system  design  time.  Thereby,  new 
stimuli  are  generated  for  the  cognitive  loop,  which 
starts  again  and  copes  with  the  crews  action.  The 
feedback  via  these  stimuli  creates  a kind  of  implicit 
communication. 

CAMA  - The  Prototype  of  a Cognitive  Assistant 
System: 

Military'  transport  missions  put  great  demands  on  the 
crew.  The  typical  scenario  is  composed  of  IFR  and 
tactical  flight  sections,  as  shown  in  figure  4. 


Figure  4:  Mission  Profile 

While  flying  IFR,  the  aircraft  may  operate  in  a high 
density  airspace.  Separation  to  other  aircraft  has  to  be 
ensured.  The  tactical  scenario  is  entered  via  a transition 
corridor.  Constraints  in  time  and  space  have  to  be  met. 
Tactical  flying  will  be  mostly  low  level  flying,  using 
terrain  masking,  even  under  adverse  weather 
conditions.  Additionally,  the  scenario  changes  at  a high 
rate  along  with  quick  reaction  required  at  times. 
Concerning  these  conditions,  technical  cognitive 
assistance  for  the  flight  crew  seems  to  be  very 
promising. 

Therefore,  the  German  DoD  started  a program,  called 
CAMA  (Crew  Assistant  Military  Aircraft),  in  order  to 
have  demonstrated  the  power  of  cognitive  automation 
for  transport  missions.  CAMA  as  a prototype  cognitive 


assistant  system  has  been  developed  by  the  University' 
of  German  Armed  Forces  Munich  in  close  cooperation 
with  DaimlerChryslcr  Aerospace,  ESG 
(Elektroniksystem  und  Logistik  GmbH)  and  DLR 
(Deutsches  Zentrum  fur  Luft-  und  Raumfahrt)  (see  [7] 
[8]). 

Structure  and  Functionality  of  CAMA: 

The  Crew  Assistant  Military  Aircraft  provides 
functionalities  in  compliance  with  many  parts  of  the 
cognitive  loop.  Again  as  depicted  in  figure  5 the 
system  is  embedded  in  the  real  world  environment. 
Information  about  this  environment  can  be  perceived 
by  means  of  sensors  and  data  links. 

The  outer  layer  of  CAMA  performs  perception  and 
interpretation  of  the  relevant  situation  elements  of  the 
real  world.  The  process  of  environment  interpretation 
as  well  as  the  interpretation  of  the  aircraft  state 
provides  information  about  the  actual  weather,  the 
proximity  to  the  terrain,  other  aircraft,  as  well  as  the 
current  state  of  aircraft  subsystems.  Tactical 
information  which  consists  of  the  mission  task,  ingress 
and  egress  corridors  and  actual  threat  situation  may  be 
fed  into  the  system.  Additionally,  data  from  computer 
vision  systems  are  included  for  machine  perception  of 
relevant  obstacles  like  landing  strips  and  obstacles  on 
uncontrolled  strips  under  low  visibility  conditions. 

All  these  pieces  of  relevant  information  are  put 
together  to  form  a central  situation  representation  that 
provides  all  the  data  which  other  CAMA  modules 
might  need  or  which  are  produced  for  further 
processing  like  the  evaluation  and  the  interpretation 
of  the  pilot’s  action.  This  core  element  of  CAMA 
forms  a close  functional  relationship  with  the  inner 
functional  layer  of  the  system  for  diagnosis  and 
detection  of  conflicts  and  opportunities.  The  elements 
of  the  central  situation  representation  that  represent  the 
relevant  objects  of  the  real  world  are  evaluated  against 
the  expected  behaviour  of  the  pilot,  the  predicted  state 
of  the  aircraft  and  against  the  overall  mission 
objectives. 


In  order  to  monitor  the  pilot’s  behaviour  the  assistant 
system  needs  a representation  of  the  expected  pilot 
actions.  In  CAMA  a normative  model  [9]  describes  the 
pilot’s  behaviour  close  to  that  as  documented  in 
handbooks  and  air  traffic  regulations.  An  adaptive 
model  [10]  covers  behavioural  traits  of  the  individual 
pilot  flying.  If  the  actual  pilot  behaviour  differs  from 
the  internal  representations  of  CAMA  then  it  can  be 
classified  into  either  errors  or  intents  (see  [II]  [12]) 
This  classification  is  based  on  the  representation  of  the 
mission  objectives  and  flight  plan  goals.  These  can  be 
explicitly  stated  by  the  pilot  as  inputs  via  the  MMI  or 
can  be  implicitly  contained  in  the  pilot’s  intent  which  is 
continuously  monitored  by  CAMA. 

If  the  pilot  behaviour  is  classified  as  an  error  a warning 
message  is  generated  and  a corrective  action  is 
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proposed  to  the  pilot.  Upon  a detected  intent  the 
internal  plan  is  adapted  accordingly.  Thus  an  implicit 
communication  between  the  pilot  and  the  system  takes 
place,  which  allows  the  pilot  to  react  to  the  current 
situation  without  having  to  tell  the  system  explicitly. 

Environment 


derived  from  the  mission  order  (e.g.  entrance  corridors 
to  gaming  area,  drop-point,  time  over  target  etc.).  A 
‘takeoff  to  landing’  mission  flight  plan  is  then 
generated.  The  1FR  flight  plan  as  part  of  it,  for 
example,  includes  the  lateral  flight  path  segments,  the 
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Figure  5:  Functional  structure  of  CAMA 


In  case  of  a possible  traffic  conflict,  for  example, 
CAMA  detects  that  the  actual  behaviour  does  not 
comply  with  the  ‘safety’  objective  and  issues  visual 
and  acoustic  advice  as  part  of  the  Traffic  Alert  and 
Collision  Avoidance  Systems  (TCAS). 

Ground  proximity  is  continuously  monitored. 
Therefore,  all  possible  flight  trajectories,  achievable  by 
full  exploitation  of  the  aircraft  performance 
capabilities,  are  checked  for  terrain  avoidance  (using  a 
Digital  Elevation  Data  database).  Again  a warning  is 
given,  visual  and  by  voice.  In  addition  an  evasive 
trajectory  is  generated. 

CAMA  also  generates  proposals  for  acting  as  part  of 
the  conflict  resolution  which  involves  planning  and 
decision  making  support.  This  functionality  ranges 
from  very  short  term  planning  e.g.  collision  and  terrain 
avoidance  to  long  term  strategic  planning.  This  enables 
the  assistant  system  not  only  to  detect  the  possible 
conflict,  but  to  generate  a conflict  solving  strategy. 
Again  all  relevant  data  needed  is  passed  over  from  the 
situation  representation  module.  In  case  of  overall 
flight  planning  all  accessible  information  about  the 
flight  is  passed  to  the  mission  planner.  This  includes 
mission  oriented  goals  and  constraints  that  can  be 


vertical  profile,  time  estimates  and  fuel  planning  [13] 
as  well  as  information  from  a navigational  database. 
Mission  constraints  which  may  change  during  the  flight 
(e.g.  a changed  exit  corridor  from  gaming  area)  or  ATC 
instructions  are  considered  during  the  planning  process. 
If  the  mission  order  leads  into  an  area  with  hostile 
radar  coverage,  the  Low  Altitude  planner  (see  [14] 
[15])  is  started  accordingly,  generating  a minimum  risk 
route  with  a maximum  probability  of  survival  in  a 
hostile  environment.  This  is  achieved  by  avoiding 
threatened  areas  if  possible,  minimizing  the  exposure 
to  unknown  threats  and  keeping  the  aircraft  clear  of 
terrain.  Therefore  the  mission  constraints,  the  tactical 
elements  and  the  resulting  threat  map,  the  terrain 
elevation  data  and  the  aircraft  performance  data  are  all 
taken  into  account.  The  generated  routes  are  passed  to 
the  crew  and  are  being  accepted  from  them  , modified 
or  rejected  respectively. 

The  calculation  is  done  in  terms  of  only  a few  seconds, 
always  giving  the  pilot  an  idea  of  what  would  be  a 
good  plan  in  the  current  situation. 

The  advanced  functionality  of  CAMA  requires  a 
sophisticated  user  interface  to  let  the  pilot  make 
advantage  of  the  system  capabilities.  Care  has  to  be 
taken  in  the  design  of  the  MMI,  not  to  produce  an  extra 
cognitive  workload. 
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Figure  6:  CAMA  Nav-Display  with  Terrain  and 
Tactical  Elements 

The  Dialogue  Management  module  [18]  of  CAMA 
ensures  that  the  communication  is  provided  to  support 
situation  awareness  in  the  best  way  possible  in  all  flight 
conditions.  It  is  based  on  the  multimodal  approach, 
which  means  that  all  pilot  inputs  can  be  performed  by 
speech,  touch-sensitive  screens  as  well  as  conventional 
line  select  keys  or  switches.  Output  makes  use  of  the 
currently  available  display  technology  and  is  presented 
by  means  of  three  high  resolution  color  displays. 
Speech  output  is  used  in  parallel  to  textual  messages. 
The  simple  graphical  user  interface  delivers  a good 
usability  already  after  a short  introduction  to  the 
system. 

Pilot  inputs  can  be: 

Request  of  flight  planning  actions 
Activation,  modification  or  rejection  of  proposals 
Activation  of  actions  related  to  warnings 
Retrieval  of  information 
Autopilot  operations 
Configuration  of  the  MMI 
Radio  management 
CAMA  outputs  can  be: 

Presentation  of  calculated  flight  plan  proposals  in 
graphical  as  well  as  textual  form 
Situation  presentation  including  tactical  and  threat 
information 

Warnings  about  detected  conflicts 
Recommendation  about  explicit  actions 
Messages  in  reply  to  requests 
Acknowledgement  of  speech  input 
Presentation  of  complex  actions  like 
briefings,  checklists  etc. 

Several  MMI  devices  provide  support  for  the  flight 
guidance  task.  For  low  level  flying  under  difficult 
weather  conditions  the  primary  flight  display  can  be 
switched  to  a 3-dimensional  presentation  of  the 
surrounding  environment  [13]. 


Results  of  Simulator  Flight  Trials: 

CAMA  is  integrated  in  the  flight  simulator  of  the 
University  of  the  German  Armed  Forces,  Munich.  This 
simulator  provides  a wide  field  of  view  visual 
simulation.  It  is  based  on  digital  terrain  and  feature 
data  and  shows  objects  like  rivers,  streets,  railroads  and 
powerlines  which  makes  it  suitable  for  low  level  flight 
simulations  based  on  terrestrial  navigation.  Three  high 
resolution  colour  monitors  with  touch-overlay  are  used 
to  display  CAMA  outputs.  Also  a number  of  realistic 
flight  controls  are  available,  including  a throttle  box, 
flaps,  gear  and  spoiler  levers,  as  well  as  an  Airbus-type 
flight  control  unit  for  autopilot  functions.  All  controls 
can  be  actively  driven  by  CAMA  on  request  of  the 
Dilot. 


In  November  1997  and  May  1998  flight  simulator  test 
runs  were  conducted  (see  [17]).  10  German  Airforce 
transport  pilots  (Airlifter  Wing  61,  Landsberg)  were 
participating  as  test  subjects.  The  pilots  were  tasked 
with  full  scale  military  air  transport  missions.  This 
comprised  a mission  briefing  with  following  takeoff 
from  base,  an  1FR  leg  to  the  ingress  corridor  and  a low 
level  flight  to  a drop  zone.  The  flight  over  hostile  area 
contained  a dynamic  tactical  scenario  with  multiple 
SAM  stations  (Surface  to  Air  Missiles).  After  the  drop 
was  accomplished  the  flight  was  led  to  the  egress 
corridor,  followed  by  an  IFR  flight  segment  to  the 
home  base. 

Each  subject  had  to  perform  the  mission  three  times. 
There  was  not  much  time  needed  for  familiarisation 
and  training  on  the  system. 

To  set  up  an  realistic  level  of  workload  several  scenario 
items  were  put  in  the  missions,  which  required  an 
action  by  the  pilot. 

The  IFR  segment  incorporated: 

Adverse  weather  conditions 

High  density  airspace  (Other  aircraft  crossing  the 
own  flight  path) 

Changing  availability  of  landing  sites 
ATC  communication  (e.g.  clearances,  radar- 
vectoring, redirection) 

The  tactical  segment  incorporated: 

Varying  SAM  sites 
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Drop  procedure 
Changed  egress  corridor 
Redirect  to  new  destination 

All  ratings  were  given  within  a range  from  1 to  7, 
where  1 represented  the  best  and  7 the  worst  score.  A 
choice  of  the  results  is  shown  in  figure  8,  9 and  10, 
where  the  ratings  are  numbered  due  to  the  order  of  test 
runs. 

A detailed  and  complete  documentation  of  the  test  runs 
and  its  results  is  given  in  [17]. 


(a)  I always  understood  CAMA's  actions 


1 2 3 4 5 6 7 


(b)  I was  (made)  aware  of  my  own  faults 


1 2 3 4 5 6 7 

Figure  8:  Evaluation  of  the  Cooperative  Approach 
of  CAM  A 
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Figure  9:  Acceptance  of  CAMA  by  Pilots 


an  assistant  system  like  CAMA.  Like  the  test  subjects 
stated,  CAMA: 

increases  Flight  Safety 
increases  Mission  Efficiency 
increases  Survival  Probability 

A more  objective  analysis  of  the  flight  simulator  trials 
was  done  by  [20]  using  an  eye  tracking  system  and  a 
data  recording  tool.  More  information  on  this  topic  can 
be  found  in  the  respective  paper  in  the  same 
proceeding. 

Actual  research: 

'■"mm  n ' - ■■vrvv^,; 


Recently  CAMA  was  being  integrated  in  the  in-flight 
simulator  ATTAS  of  the  DLR  (shown  in  figure  1 1)  and 
was  successfully  tested  and  demonstrated  in  several 
flight  experiments  in  March  2000.  Further  trials  are 
scheduled  for  November  2000.  These  flight  tests 
comprised  IFR  and  low  level  flight  segments  as  they 
occur  in  a military  air  transport  mission.  Again  subjects 
were  experienced  air  transport  pilots  from  the  German 
Air  Force.  Data  from  sensor  input  as  well  as  the 
internal  system  states  were  recorded,  which  will  allow 
a replay  of  the  conducted  flights  in  the  experimental 
flight  simulator  at  the  university  of  armed  forces  in 
Munich. 


(a)  CAMA  increases  Flight  Safety 

ill!!!! 

(b)  CAMA  increases  Mission  Efficiency 


1 2 3 4 5 6 7 

(cj  CAMA  increases  Survival  Probability 


1 2 3 4 5 6 7 

Figure  10:  Overall  Evaluation  of  CAMA 


Especially  the  rating  concerning  the  overall  evalutaion 
as  shown  in  figure  10  points  out  clearly  the  benefits  of 


Conclusion:  Future  battlefield  scenarios  will  be 
characterised  by  the  availability  of  a greater  amount  of 
information.  Onboard  infonnation  processing  puts 
great  demand  on  the  aircrew,  which  may  lead  to 
overcharging  of  the  crew. 

To  cope  with  these  changing  conditions,  the  approach 
of  a cognitive  assistant  system  was  investigated.  It 
offers  support  to  the  aircrew  regarding  enhancement  of 
situation  awareness,  handling  of  multifunctional  tasks 
and  situation-dependent  balancing  of  workload  for  the 
sake  of  mission  effectiveness  and  safety.  It  has  become 
increasingly  evident  that  this  cannot  be  achieved 
without  moving  towards  the  cognitive  approach. 

The  presented  approach  and  its  realisation  in  the 
prototype  system  CAMA  has  been  demonstrated.  The 
benefits  are  already  demonstrated  in  the  course  of 
simulator  trials  and  In-flight  demonstrations. 
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